Iterative Reweighted Least Squares ∗

Describes a powerful optimization algorithm which iteratively solves a weighted least squares approximation problem in order to solve an L_p approximation problem. 1 Approximation Methods of approximating one function by another or of approximating measured data by the output of a mathematical or computer model are extraordinarily useful and ubiquitous. In this note, we present a very powerful algorithm most often called Iterative Reweighted Least Squares or (IRLS). Because minimizing the weighted squared error in an approximation can often be done analytically (or with a nite number of numerical calculations), it is the base of many iterative approaches. To illustrate this algorithm, we will pose the problem as nding the optimal approximate solution of a set of simultaneous linear equations  a11 a12 a13 · · · a1N a21 a22 a23 a31 a32 a33 .. .. aM1 · · · aMN   x1 x2 x3 .. xN  =  b1 b2 b3 .. bM  (1) or, in matrix notation Ax = b (2) where we are given an M by N real matrix A and an M by 1 vector b, and want to nd the N by 1 vector x. Only if A is non-singular (square and full rank) is there a unique, exact solution. Otherwise, an approximate solution is sought according to some criterion of approximation. If b does not lie in the range space of A (the space spanned by the columns of A, [8]), there is no exact solution to (2), therefore, an approximation problem is posed to be solved by minimizing the norm (or some other measure) of an equation error vector de ned by e = Ax− b. (3) A more general discussion of the solution of simultaneous equations can be found the the Connexions module [8]. ∗Version 1.12: Dec 17, 2012 2:09 pm -0600 †http://creativecommons.org/licenses/by/3.0/ http://cnx.org/content/m45285/1.12/ OpenStax-CNX module: m45285 2 2 Least Squared Error Approximation A generalized solution (an optimal approximate solution) to (2) is usually considered to be an x that minimizes some norm or other measure of e. If that problem does not have a unique solution, further conditions, such as also minimizing the norm of x, are imposed and this combined problem always has a unique solution. The l2 or root-mean-squared error or Euclidean norm is √ eTe and its minimization has an analytical solution. This squared error is de ned as